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Abstract 


This study investigated the relationships between social and emotional learning factors and 
course failure for 9,308 7th—-9th graders. Student self-report measures of social and emotional 
learning (SEL) and biographical factors (ACT® Engage® Grades 6-9) were analyzed through 
multiple regression models with propensity score weighting techniques. The results showed 
that given a sample of 7th graders, the increase in academic discipline, optimism, and orderly 
conduct, as well as having parents/guardians with higher educational degrees and expected 
educational degree from family could reduce the likelihood of course failure. Further, students 
who reported failing a course in grade 7 demonstrated lower SEL factors in grade 8; similarly, 
students failing a course in grade 8 demonstrated lower SEL factors in grade 9. Finally, for 7th 
graders reporting a failed class within the last year, an increase in SEL factors was shown to 
prevent additional course failures in grades 8 and 9. Interventions helping students who have 
failed a class to develop their SEL skills and how SE information can be used in early warning 
systems to help identify at-risk students are discussed. 


Introduction 


Middle school is a key transition period for the development of effective behaviors that can 
facilitate later high school and postsecondary academic success (ACT, 2008). Surprisingly, 
statistics of national or state course failure rates in secondary school are rarely released to the 
public. One exception comes from Washington state, which, based on its 2016 data, reported 
that 23% of grade 9 students failed at least one core course in English, math, or science (state 
of Washington, 2017). Other research shows that course failure has a large impact on likelihood 
of graduation (Allensworth & Easton, 2007; Balfanz, Herzog, & Mac Iver, 2007), which has led to 
course failure being an important indicator in early warning systems for middle and high school 
dropout. 


Casillas et al. (2012) suggested that psychosocial factors (a.k.a. social and emotional learning 
[SEL] factors) are related to subsequent academic outcomes and course failure that may later 
result in dropout. However, the study did not show which SEL factors have more predictive 
power than others in predicting course failure. The current study extends previous work by 
investigating the relationships between course failure, SEL, and biographical factors. The 
research questions include: 


1. Which SEL and biographical factors are most predictive of failing a class? 


2. Does failing a class predict subsequent changes in SEL factors? 


3. Which SEL and background factors predict recovery from failing a class? 
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Method 


Participants 


A total of 9,308 students were included in the study based on two criteria: (a) students took an 
assessment of SEL factors (ACT Engage Grades 6-9) annually in grades 7, 8, and 9; and (b) 
students’ responses in the assessment were not flagged due to item response problems (e.g., 
response inconsistency, missing data). 


Measures 


SEL factors. Consistent with earlier studies (Casillas et al., 2012), measures of SEL include: 
Academic Discipline, Commitment to School, Optimism, Family Attitude toward Education, 
Family Involvement, Relationships with School Personnel, School Safety Climate, Managing 
Feelings, Orderly Conduct, and Thinking before Acting. The 10 scales show moderate to high 
internal reliabilities (a = .87 - .90, Mdn = .87). Scale definitions, sample items, and psychometric 
properties are presented in the Appendix (ACT, 2011). 


Biographical factors. Included biographical variables consisted of: age, gender, primary 
language spoken, the highest parent/guardian education, the highest expected educational 
degree from family, and hours spent on watching TV, playing video games, and surfing the 
Internet for materials not related to schoolwork on a school day. 


Course failure. Course failure was reported by students through their response to this question: 
“| have failed a class within the last year.” 


Data Analyses 


Multiple logistic regression models were used to address research questions 1 and 3. Separate 
models were fit for each of the 10 measures of SEL. For research question 1, the dependent 
variable is whether students reported failing a class within the last year as a 7th grader (Yes 
was coded as 1, and No was coded as 0), while the independent variables include gender, 
primary language spoken, age, the SEL measure from grade 7, and biographical factors. 

For research question 3, the dependent variable is “recovery from course failure,” which is 
defined as reporting no failed classes as an 8th grader and as a 9th grader. The models for 
research question 3 were only fit for students who reporting failing a class in grade 7, and the 
independent variables are the same variables described in research question 1 and grade 7 
course failure propensity score (described below). The SEL measure and factors such as hours 
spent on watching TV, playing video games, and surfing the Internet are the means from grades 
8 and 9. 


Research question 2 was addressed by two sets of multiple regression analyses when 

students were in grade 7 and grade 8, respectively. Each set includes a separate multiple linear 
regression model for each of the 10 SEL measures. Regarding the first set of analyses, the 
dependent variable is the SEL measure from grade 8 (e.g., Academic Discipline in grade 8) 

and independent variables include the SEL measure from grade 7 (e.g., Academic Discipline 

in grade 7), grade 7 course failure propensity score, and course failure reported in grade 7. 
Similarly, the dependent variable for the second set of analyses is the grade 9 SEL measure 
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and independent variables include the grade 8 SEL measure, grade 8 course failure propensity 
score, and course failure reported in grade 8. Figure 1 summarizes the research questions and 
variables. 


The propensity score is a conditional probability proposed by Rosenbaum and Rubin (1983) that 
“aggregates a number of characteristics that individually would be difficult to match among those 
in the treatment and nontreatment groups” (Schneider, Carnoy, Kilpatrick, Schmidt, & Shavelson, 
2005, p. 49). For research question 2, we are interested in the effect of failing a class on future 
measures of SEL factors. Because students who fail a class and students who do not fail a class 
are different on many baseline variables (e.g., biographical, and prior SEL measures), we used 
propensity scores to adjust for the baseline differences. Grade 7 course failure propensity score 
is the predicted probability of grade 7 course failure (derived from a logistic regression model), 
conditional on grade 7 SEL measures, biographic factors, and grades from the past two years. 


In addition, according to the recommendations of Aiken and West (1991), the SEL scores 
and biographical factors (e.g., highest expected educational degree from family, hours spent 


watching TV, playing video games, and surfing the Internet) were transformed to z-scores for the 
regression analyses. 


Figure 1.A Path Diagram Representing the Research Questions and Variables 


RQ3 


RQ3 
SEL —> SEL —> SEL 
Biographic a Recovery from 
variables RQ2 pn ws“ course failure 
RQ1 
Course Course Course 
failure failure failure 
7 8 9 


Grade Level 
Results 


Research Question 1: SEL and Biographical Factors in 
Predicting Course Failure 


In Table 1, the model shows that the odds of reporting class failure at the first wave of data (7th 
grade) decreases by 51%, 8%, and 37% for every standard deviation increase in Academic 
Discipline, Optimism, and Orderly Conduct, respectively, even when controlling for other 
variables. Regarding biographical factors, the model shows that the odds of failing a class 
decrease by 16% and 8% for every standard deviation increase in the highest parent/guardian 
education and the highest expected educational degree. 
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Table 1. Relationship between Course Failure Experience, SEL and Biographical Factors 


Odds Ratio 
WEVar-1e) (1-9 B Standard Error => qo (6) 
Intercept -0.882*** 0.069 0.414 
Male 0.078 0.058 1.082 
Language 0.054 0.054 1.056 
Age OAS 0.026 1.110 
Parent’s Highest Educational Degree -0.177*** 0.028 0.838 
Parent’s Highest Educational Degree (unknown) -0.002 0.087 0.998 
Academic Discipline -0.708*** 0.042 0.493 
Commitment to School 0.078 0.040 1.081 
Family Attitude toward Education 0.045 0.041 1.046 
Family Involvement 0.043 0.039 1.044 
Managing Feelings 0.044 0.041 1.045 
Optimism -0.086* 0.037 0.918 
Orderly Conduct -0.461*** 0.037 0.631 
Relationships with School Personnel -0.018 0.035 0.982 
Thinking before Acting 0.083* 0.039 1.086 
School Safety Climate 0.037 0.033 1.037 
Highest expected educational degree from family -0.080** 0.027 0.923 
Hours spent watching TV 0.013 0.028 1.013 
Hours spent playing video games -0.016 0.030 0.984 
Hours spent surfing the Interneta -0.004 0.028 0.996 


Note. n = 8,584. A multiple logistic regression model was constructed. Dependent variable refers 
to Grade 7 students who failed a class within the last year or not (Yes= 1; No = 0). 

a Surf the Internet for materials not related to schoolwork on a school day. 

*p < .05 (two-tailed). ** p < .01 (two-tailed). *** p < .001 (two-tailed). 


Research Question 2: Changes in SEL Factors Related to Course 
Failure 


In Table 2, students who reported failing a course within the past year in grade 7 scored lower in 
Academic Discipline, Family Attitude, Family Involvement, Managing Feelings, Orderly Conduct, 
and Thinking before Acting in grade 8. Similarly, students who reported failing a course in grade 

8 scored lower in Academic Discipline and Commitment in grade 9. Take Academic Discipline in 
grade 8 as an example. The results show that past course failure experiences reported in grade 7 
are negatively related to Academic Discipline in grade 8 (B = -0.082, p < .001), after controlling for 
Academic Discipline in grade 7 and grade 7 course failure propensity score. Failing a course was 
related to a small decrease (0.082 SD units) in Academic Discipline in grade 8. 
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Table 2. Prediction of Changes in SEL Factors 


eX=1 a (ore | SEL Factor AYE-Ta 10) (1-3 ele] 

Error 

Intercept 0.312*** 0.028 

ere Academic Discipline 0.482*** 0.014 
Academic Discipline . 

Propensity Score -0.820*** 0.064 

Course Failure -0.082*** 0.020 

Intercept OM/Gnas 0.021 

C it tie ernoa Commitment to School 0.485*** 0.010 

ommitment to Schoo 

Propensity Score -0.481*** 0.049 

Course Failure -0.032 0.022 

Intercept 0.201*** 0.023 

Family Attitude toward Family Attitude 0.501*** 0.010 

Education Propensity Score -0.548*** 0.048 

Course Failure -0.045* 0.022 

Intercept One 0.021 

: Family Involvement OP56 Gnas 0.010 
Family Involvement : 

Propensity Score -0.420*** 0.047 

Course Failure -0.042* 0.021 

Intercept 0.195*** 0.027 

; : Managing Feelings OoTT* 0.010 
Managing Feelings : 

Propensity Score -0.518*** 0.048 

G7-G8° Course Failure -0.067*** 0.020 

‘ Intercept Oma Sans 0.022 

ae Optimism (OP Silane 0.010 
Optimism : 

Propensity Score -0.417*** 0.050 

Course Failure -0.040 0.022 

Intercept 0.2017" 0.025 

Orderly Conduct 0.556*** 0.012 
Orderly Conduct ; 

Propensity Score -0.527*** 0.056 

Course Failure -0.086*** 0.020 

Intercept Oleg 0.029 

Relationships with School Relationships with School Personnel ORS aes 0.010 

Personnel Propensity Score 0.423" 0.048 

Course Failure -0.015 0.022 

Intercept 0.087* 0.039 

; School Safety Climate 0.516*** 0.010 
School Safety Climate ; 

Propensity Score -0.130** 0.046 

Course Failure -0.002 0.022 

Intercept 0.254*** 0.023 

=e ; Thinking before Acting OI524555 0.010 
Thinking before Acting ; 

Propensity Score -0.677*** 0.050 


Course Failure -0.069*** 0.021 
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Table 2. Prediction of Changes in SEL Factors (continued) 


Period SEL Factor AYE-Ta- 10) (-1-] B Staal 

ace) g 

Intercept -0.112*** 0.025 

Ae alee Academic Discipline 0.536*** 0.016 
Academic Discipline ; 

Propensity Score -0.122*** 0.012 

Course Failure -0.050* 0.023 

Intercept -0.099*** 0.026 

J Commitment to School 0.486*** 0.012 
Commitment to School ; 

Propensity Score -0.106*** 0.010 

Course Failure -0.065* 0.026 

Intercept -0.098*** 0.024 

Family Attitude toward Family Attitude 0.518*** 0.012 

Education Propensity Score -0.095*** 0.009 

Course Failure -0.034 0.025 

Intercept -0.046* 0.019 

J Family Involvement OlGS Sinan 0.011 
Family Involvement ; 

Propensity Score -0.059*** 0.009 

Course Failure -0.035 0.023 

Intercept -0.084*** 0.022 

. ; Managing Feelings 0.626*** 0.011 
Managing Feelings . 

Propensity Score -0.088*** 0.009 

Course Failure -0.035 0.022 

Intercept -0.050* 0.021 

ee Optimism Or 0.011 
Optimism ‘ 

Propensity Score ORO Sims 0.009 

Course Failure -0.030 0.025 

Intercept -0.090*** 0.022 

Ben cond Orderly Conduct 0.564*** 0.014 

rder ndu 

sik aaa Propensity Score -0.091*** 0.011 

Course Failure -0.037 0.023 

Intercept -0.021 0.027 

Relationships with School Relationships with School Personnel OlGMO5s5 0.011 

Personnel Propensity Score 20.050" 0.009 

Course Failure -0.018 0.024 

Intercept 0.026 0.049 

: School Safety Climate 0.574*** 0.011 
School Safety Climate ; 

Propensity Score -0.020* 0.009 

Course Failure -0.017 0.024 

Intercept -0.075** 0.021 

Pe ; Thinking before Acting OlG2ae 0.011 
Thinking before Acting : 

Propensity Score -0.075*** 0.009 

Course Failure -0.016 0.023 


Note. n = 8,042 and n = 6,300 in predicting SEL skills in grade 8 and grade 9, respectively. 

a Dependent variables are measured in grade 8, while independent variables are measured 
in grade 7. b Dependent variables are measured in grade 9, while independent variables are 
measured in grade 8. 


* pb < .05 (two-tailed). ** p < .01 (two-tailed). *** p < .001 (two-tailed). 
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Research Question 3: SEL and Biographical Factors in Predicting 
Recovery from Course Failure 


In Table 3, the model shows that, for 7th graders reporting a failed class within the last year, the 
odds of recovery from course failure (i.e., no failed classes as 8th graders and as 9th graders) 
increase by 224%, 121%, and 153% for every standard deviation increase in Academic Discipline, 
Optimism, and Orderly Conduct, respectively. 


Table 3. Prediction of Recovery from Course Failure Experience 


J tclaler-lae Mumm @Lo(e (mae: lilo) 


AYE-Ta-] 0) (-X-) B Error => Con (6) 
Intercept -0.453* 0.194 0.636 
Propensity Score -0.835* 0.341 0.434 
Male -0.164 0.143 0.849 
Language 0.04 0.130 1.041 
Age -0.008 0.062 0.992 
Parent’s Highest Educational Degree 0.106 0.068 2 
Parent’s Highest Educational Degree (unknown) -0.124 0.249 0.883 
Academic Discipline 0.808*** 0.115 2.243 
Commitment to School -0.012 0.108 0.988 
Family Attitude toward Education -0.116 OMS 0.890 
Family Involvement 0.002 0.098 1.002 
Managing Feelings -0.173 0.103 0.841 
Optimism 0.188* 0.093 1.207 
Orderly Conduct 0.424*** 0.098 1.528 
Relationships with School Personnel 0.041 0.092 1.042 
Thinking before Acting -0.096 0.101 0.908 
School Safety Climate -0.114 0.077 0.892 
Highest expected educational degree from family 0.021 0.066 1.021 
Hours spent watching TV 0.002 0.066 1.002 
Hours spent playing video 0.002 0.071 1.002 
Hours spent surfing the Internet -0.106 0.070 0.899 


Note. n = 1,785. A multiple logistic regression model was constructed. The participants in the 
analysis are students who failed a class in grade 7. Dependent variable refers to students who do 
not fail a class in grades 8 and 9 (Yes= 1; No = 0). 

“p< .05 (two-tailed). ™ p < .001 (two-tailed). 
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Discussion 


This longitudinal study shows that several SEL and biographical factors are associated with 
course failure and that course failure is associated with later negative changes in SEL factors 
during secondary school. However, the results also show that, by developing some of these 
factors after the initial course failure, it is likely that students could prevent subsequent academic 
difficulties, additional course failures, and the drop in some SEL factors (e.g., self-esteem) that 
happens after course failure. 


A foundational level of knowledge and SEL skills are required for student success in secondary 
school (e.g., Casillas, Robbins, Allen, Kuo, Hanson, & Schmeiser, 2012; Kuo, Casillas, & Allen, 
2019). To improve SEL skills, effective interventions that integrate SEL knowledge and skills 

into existing curriculum are suggested. Studies on the efficacy of behavioral skill intervention 
should make use of SAFE programming. SAFE stand for Sequenced (programs with a connected 
and coordinated set of activities), Active (active forms of learning), Focused (programs having 
components to develop personal or social skills), and Explicit (programs targeting specific skills; 
Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, 2011A more compact intervention framework 
emphasizing positive behavioral interventions and supports may develop student behavioral skills 
(Bradshaw, Bottiani, Osher, & Sugai, 2014). 


In addition to existing early warning flags (e.g., low attendance rate, poor academic achievement, 
and suspensions) that focus educators on students at risk of dropping out (Balfanz et al., 2007), 
this study suggests that educators use measures of SEL skills in early warning systems. These 
skills can help identify students at risk of failing courses and assist them before they experience 
negative academic outcomes. Such intervention can prevent further increasing the gap between 
high- and low-achieving students. 


Among the limitations of the study is that course failure was reported by students and was limited 
to a single binary response. By not examining both number and types of courses, we may be less 
likely to detect differential effects. Another possible limitation is that when 7th graders responded 
to the item: “I have failed a class within the last year,” they may consider their grades in either 
grade 6 or grade 7 depending on their testing time. That is, if they took the assessment in the 
spring semester of 7th grade, they may consider their grades in the fall semester of 7th grade or 
in the spring semester of 6th grade. 
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Appendix 


Table A1. ACT Engage Scales, Definitions, and Sample Items 


# of 


Academic 
Discipline 


Commitment to 
School 


Optimism 


Family Attitude 
toward Education 


Family 
Involvement 


Relationships 
with School 
Personnel 


School Safety 
Climate 


Managing 
Feelings 


Orderly Conduct 


Thinking before 
Acting 


Definition 
Degree to which a student 
is hardworking and 
conscientious as evidenced 
by the amount of effort 
invested into completing 
schoolwork. 


Commitment to stay in 
high school and obtain a 
diploma. 


Having a hopeful outlook 
about the future in spite of 
difficulties or challenges. 


Positive family attitude 
regarding the value of 
education. 


Family involvement ina 
student’s school life and 
activities. 


The extent to which 
students relate to school 
personnel as part of their 
connection to school. 


School qualities related 
to student's perception of 
security at school. 


Tendency to manage 
duration and intensity of 
negative feelings, and to 
find appropriate ways to 
express these feelings. 


Tendency to behave 
appropriately in class and 
avoid disciplinary action. 


Tendency to think about 
the consequences of one’s 
actions before acting. 


Sample items 


1. | turn in my assignments on time. 


2. I’m usually prepared for class. 


1. |am committed to graduating from 
high school. 


2. Ahigh school diploma is important for 
getting ahead in life. 


1. | am confident that everything will turn 
out all right. 


2. | look for the bright side of things. 

1. My family supports my efforts in 
school. 

2. Education is important to my family. 
1. My family tries hard to be involved in 
my school life. 

2. | talk to my family about school 
accomplishments. 

1. Adults at my school understand point 
of view. 

2. If | was in trouble, adults at my school 
would be there to help. 

1. | feel safe at school. 

2. Students at my school don’t get in 
trouble with the law. 

1. | would walk away if someone wanted 
to fight me. 

2. | fight back when someone makes me 
mad (reverse-keyed). 

1. | joke around or make fun of others 
during class. 

2. | have been sent to the principal's 
office for misbehaving (reverse-keyed). 
1. | think about what might happen 
before | act. 


2. | think about what | say before | say it. 


items 


11 


10 


10 


10 


2 


11 


12 


12 


Note. Adapted from “ACT ENGAGE Grades 6-9 User’s Guide,” by ACT, 2011. Copyright 2011 by ACT. 


a-scale 


.90 


85 


.88 


84 


87 


.89 


81 


.89 


.80 


.86 
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